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EVALUATION  OF  THE  TRILAB  SYSTEM:  EXECUTIVE  SUMMARY 


A.  INTRODUCTION 

- -^The  military  services,  with  the  coordination  of  The  Tri-Service 

Medical  Information  (TRIMIS)  Program  Office  (TPO) ,  have  installed 
computerized  medical  laboratory  systems  (TRILAB)  in  three  military 
hospitals : 

^  Naval  Regional  Medical  Center  (NRMC) ,  Oakland; 

y  Dwight  D.  Elsenhower  Army  Medical  Center  (DDEAMC), 

Fort  Gordon,  Georgia;  and 

Wright  Patterson  Medical  Center  and  Regional  Hospital 
(Wright  Patterson),  Dayton,  Ohio. 

The  TRILAB  system  is  designed  to  support  the  following  laboratory 
activities:  patient  files,  test  order  entry,  specimen  accessioning 

and  control,  work  document  preparation,  quality  control,  test  result 
entry,  inquiry  and  test  retrieval,  test  result  reporting  at  wards  and 
clinics,  and  management  reporting.  The  system  is  designed  to  have 
automated,  high-volume  test  Instruments  on-line,  with  the  goal  of 
reducing  clerical  work  of  laboratory  technicians  and  transcription 
errors,  and  to  monitor  quality  control  samples  in  order  to  check  for 
correct  calibration  of  instruments  and  proper  handling  of  specimens 
within  the  laboratory.  The  system  produces  interim  test  result 
reports,  daily  cumulative  reports,  and  cumulative  summary  discharge 
reports.  In  addition,  the  system  produces  management  information, 
such  as  laboratory  workload  summary  reports.  The  system  supports 
terminals  outside  the  laboratory,  such  as  in  wards,  clinics,  and 
satellite  facilities,  for  transmission  of  results  and  for  inquiries  as 
to  test  status.  * 

Arthur  D.  Little,  under  contract  to  the  TPO,  conducted  an 
evaluation  of  the  TRILAB  system.  The  evaluation  was  based  primarily 
on  a  comparison  of  information  and  data  on  operations  collected  at 


NRMC  Oakland,  which  was  chosen  as  the  primary  evaluation  site  for  the 
system.  Baseline  data  were  collected  during  the  fall  of  1980,  prior 
to  implementation  of  the  TRILAB  system  which  occurred  early  in  1982. 


The  post-implementation  survey  was  carried  out  In  October  1982.  In 
addition  to  the  comprehensive  evaluation  for  NRMC  Oakland,  pre-  and 
post-implementation  surveys  were  carried  out  at  Wright  Patterson,  and 
interviews  and  Implementation  monitoring  were  performed  at  Eisenhower. 

On  July  15,  1977  the  TRIMIS  Medical  Review  Group  (MRG)  developed 
'  seven  project  objectives  for  the  Tri-Service  Laboratory  System: 

e 

e  To  make  information  available  to  physicians  with 
increased  efficiency  and  accuracy; 

e  To  present  the  data  in  a  convenient  and  meaningful  manner 
with  sufficient  variety  in  report  formats  to  meet  the 
needs  of  all  users; 

a  To  be  able  to  handle  increased  demands  for  laboratory 
tests  without  significant  increases  in  staff; 

•  To  provide  accountability  of  laboratory  requests  and  to 
monitor  generation  of  test  results  to  include  providing 
notices  of  abnormal  values  or  Improper  quality  control 
results  as  soon  as  they  are  available; 

•  To  gather,  as  a  result  of  normal  procedures,  workload  and 
managerial  data,  and  to  present  this  as  required  in  order 
to  assist  in  decision-making  in  the  laboratory; 

•  To  reduce  the  clerical  work  required  of  qualified 
technicians  in  the  laboratory; 

e  To  Improve  result  accuracy  by  eliminating  transcription, 
calculation,  and  specimen  Identification  error. 

The  next  sections  summarize  the  specific  findings  and  relate  them 
to  these  project  objectives. 

B.  PIMPINGS 

1.  Time  devoted  to  information  handling.  At  Oakland,  time  devoted  to 
information  handling  activities  in  the  post-implementation  period  was 
approximately  2.6?  lower  than  in  the  baseline  period.  Based  on 
current  staffing,  this  was  equivalent  to  a  net  reduction  of  34  hours 
per  week  (day  shift,  Monday  through  Friday)  devoted  to  information 
handling  activities  in  the  Chemistry,  Hematology  and  Microbiology 
Sections.  There  were  reductions  in  time  devoted  to  transcription  and 
recording  of  test  results,  compilation  of  workload  statistics,  and  for 
quality  control  reporting. 
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2.  Turnaround  time.  For  STAT/urgent  tests,  process  times— 'times 
between  receipt  of  requisition  and  transmission  of  test  result  to 
provider  locations  (via  telephone  in  the  baseline  period  and  via 
terminal  in  the  post-implementation  period) — were  either  unchanged 
(Chemistry)  or  reduced  (Hematology) . 

For  routine  tests,  results  were  available  to  provider  locations 
(via  terminal)  sooner  in  the  post-implementation  period  than  in  the 
baseline  period  for  Chemistry  and  Hematology,  and  in  about  the  same 
time  period  or  sooner  for  Microbiology.  Hard-copy  daily  reports  from 
the  system  were  generally  available  to  providers  later  than  the 
completed  results  requisition  slips  were  in  the  baseline  period.  The 
hard-copy  interim  reports  (for  the  surgical  floors)  were  available 
sooner  in  the  post-implementation  period  for  hematology  tests  and  in 
approximately  the  same  time  for  chemistry  and  microbiology  tests. 

3.  Number  of  telephone  calls.  Volume  of  telephone  calls  to  the 
laboratory  to  inquire  about  test  status  or  test  results  was  reduced 
considerably  at  all  three  sites  after  implementation  of  the  TRILAB 
system.  The  volume  of  telephone  calls  in  the  post-implementation 
period  was  almost  half  that  in  the  baseline  study  period. 

4.  Staff  perception  of  laboratory  services.  In  addition  to  the 
significant  reduction  in  telephone  calls,  laboratory  staff  Indicated 
that  time  spent  on  manual  record  keeping  was  reduced  as  a  result  of 
implementation  of  the  TRILAB  system.  Transcription  discrepancies  and 
repeating  of  tests  due  to  Inaccurate  results  were  judged  to  occur  with 
similar,  or  slightly  less  frequency,  as  a  result  of  Implementation  of 
TRILAB. 

Providers  showed  positive  changes  in  attitudes  toward  laboratory 
services  after  implementation  of  the  TRILAB  system.  Problem 
occurrences  were  considered  to  be  less  frequent  after  implementation 
of  TRILAB.  Providers  felt  that  tests  repeated  due  to  delays,  volume 
of  telephone  calls  to  inquire  about  test  status  and  results,  tests 
repeated  due  to  lost  results,  and  tests  repeated  due  to  filing  delays 
or  due  to  inaccurate  results  had  all  been  reduced  as  a  result  of  the 
TRILAB  installation. 


Hurting  staff  at  all  thraa  sites  est lasted  that  there  was  considerable 
reduction  In  staff  tiae  associated  with  telephoning  the  laboratory  to 
receive  test  results  and  to  Inquire  about  late  or  alsslng  results. 
Tiae  spent  in  filing  test  results  and  in  chart  review  was  reduced  due 
to  having  cuaulative  reports  available.  It  was  estimated  by  nursing 
staff  at  all  three  sites  that  savings  amounted  to  approximately  four 
hours  per  day  per  inpatient  unit  or  clinic  which  had  terminals.  These 
savings  in  staff  tiae,  which  were  made  available  for  other  activities 
and  direct  patient  care,  amounted  to  75  nursing  staff  hours  per  day  at 
NRMC  Oakland,  49  hours  per  day  at  Wright  Patterson,  and  26  hours  per 
day  at  Eisenhower. 

In  addition  to  the  quantifiable  benefits  described  above,  staff 
reported  that  morale  had  Improved  considerably  as  a  result  of  the 
reduction  in  telephone  calls  between  patient  care  units  and  the 
laboratory,  because  of  the  ability  to  look  up  test  status  via  the 
computer  terminal.  Also,  it  was  felt  that  patient  care  had  improved 
as  a  result  of  highlighting  abnormal  results  on  the  reports,  and 
improved  ability  to  obtain  results  both  on  current  tests  and  on  tests 
carried  out  previously. 

A  further  indirect  measure  of  approval  of  the  system  was  the 
expressed  dealre  of  the  staff  in  those  inpatient  units  and  outpatient 
clinics  which  did  not  have  terminals  for  a  terminal  in  their  own 
location. 

C.  COMPARISON  OF  RESULTS  WITH  PROJECT  OBJECTIVES 

It  is  concluded  that  the  original  project  goals  have  been  met  by 
the  TRILAB  system.  The  system  is  very  effective  in  making  information 
available  to  providers  with  Increased  efficiency  and  accuracy,  in 
presenting  test  result  data  in  useful  and  meaningful  format,  in 
highlighting  abnormal  values  and  monitoring  quality  control  results, 
and  in  automatically  capturing  workload  and  managerial  data.  The 
objectives  of  enabling  the  laboratory  to  handle  Increased  demands  for 
testing  and  for  reducing  the  clerical  work  required  by  laboratory 
technicians  were  also  met,  but  more  modestly. 
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I .  INTRODUCTION 


The  Tri-Service  Medical  Information  System  (TRIMIS)  Program 
Office  (TPO)  has  installed  computerized  clinical  laboratory  systems 
(TRILAB)  in  three  military  hospitals: 

•  Naval  Regional  Medical  Center  (NRMC) ,  Oakland; 

•  Dwight  D.  Eisenhower  Army  Medical  Center  (DDEAMC) , 

Fort  Gordon,  Georgia; 

•  Wright  Patterson  Medical  Center  and  Regional  Hospital 
(Wright  Patterson),  Dayton  Ohio. 

The  experience  with  these  installations  is  being  evaluated  to 
provide  information  for  use  in  decision  making  about  the  future  use  of 
automation  in  clinical  laboratories  in  other  milir-ry  health  care 
facilities. 

The  first  installation  was  scheduled  for  NRMC  ’.and,  which  was 
chosen  as  the  primary  evaluation  site  for  the  T  A.B  system.  The 
evaluation  plan  for  the  TRILAB  system  was  deve  *  .  by  Analytic 
Services,  Inc.  (ANSER)  ^  who  also  collected  the  baseline  data  at 
Oakland  during  an  eight-week  period  (September  29,  1980-November  29, 
1980) .  The  implementation  of  the  TRILAB  system  was  initiated  in  early 
1982.  The  post-implementation  survey  was  carried  out  in  October  1982 
by  Arthur  D.  Little,  Inc. 

In  addition  to  the  comprehensive  evaluation  plan  for  NRMC 
Oakland,  "mini-evaluations"  were  subsequently  planned  for  the  other 
two  sites  (Wright  Patterson  and  Eisenhower).  Baseline  data,  using 
self-administered  questionnaires,  were  collected  by  Arthur  D.  Little, 
Inc.  at  these  two  sites  in  the  fall  of  1981.  A  post-implementation 
survey  was  carried  out  at  Wright  Patterson  in  the  fall  of  1982.  The 
system  at  Eisenhower,  the  last  of  the  three  facilities  to  implement 
TRILAB,  had  not  reached  stability  and  complete  implementation  at  the 
time  of  preparation  of  this  report.  Based  on  the  findings  of 
implementation  monitoring,  it  is  anticipated  that  the  results  from 
Eisenhower  would  be  consistent  with  the  evaluation  conclusions  based 
on  the  other  two  sites..  Results  of  the  implementation  monitoring 
survey  at  Eisenhower  are  included  in  this  report. 


An  assessment  of  the  baseline  data  has  been  reported  previously 

(2) 

in  a  six-volume  baseline  report.  This  report.  Volume  I  of  a 
four-volume  report,  presents  an  overview  and  executive  summary  of  the 
evaluation  of  the  TRILAB  system.  Volume  II  presents  the  evaluation 
for  NRMC  Oakland  and  Volume  III,  the  survey  results  and  evaluation  for 
Wright  Patterson.  Volume  IV  contains  supporting  appendices  for 
Volumes  II  and  III. 

The  next  chapter  presents  a  brief  description  of  the  TRILAB 
system  and  an  overview  of  the  TRILAB  system  objectives  and  evaluation 
approach.  Chapter  III  summarizes  the  results  of  the  evaluation  at  the 
three  sites.  Chapter  IV  summarizes  the  overall  conclusions. 
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II.  DESCRIPTION  OF  TRILAB  SYSTEM  AND  EVALUATION  APPROACH 


A.  DESCRIPTION  OF  TRILAB 

The  TRILAB  system  is  designed  to  support  the  following  laboratory 
activities:  patient  files,  test  order  entry,  specimen  accessioning 
and  control,  work  document  preparation,  quality  control,  test  result 
entry,  inquiry  and  data  retrieval,  test  result  reporting  at  wards  and 
clinics,  and  management  reporting. 

The  TRILAB  system  is  designed  to  have  automated,  high-volume  test 
instruments  on-line,  with  the  goal  of  significantly  reducing  clerical 
work  of  laboratory  technicians  and  transcription  errors.  The  system 
is  also  designed  to  monitor  quality  control  samples  in  order  to  check 
for  correct  calibration  of  instruments  and  proper  handling  of  the 
specimens  within  the  laboratory,  and  to  produce  interim  test  results 
reports,  daily  cumulative  reports,  and  cumulative  summary  discharge 
reports. 

In  addition,  the  system  produces  management  information,  such  as 
laboratory  workload  summary  reports,  which  should  reduce  the  effort  to 
prepare  management  reports  and  assist  in  the  efficient  organization 
and  administration  of  the  laboratory. 

The  TRILAB  system,  which  uses  software  developed  by  Meditech,  was 
obtained  from  Centennial  Systems  Corporation  through  competitive 
procurement.  The  system  can  support  terminals  outside  the  laboratory, 
such  as  in  wards,  clinics  and  satellite  facilities,  for  transmission 
of  results  and  for  inquiry  as  to  test  status. 

B.  DESCRIPTION  OF  CLINICAL  LABORATORY  OPERATIONS 

Summary  statistics  for  the  three  evaluation  sites  are  presented 
in  Table  1.  During  FY  1982,  the  clinical  laboratory  at  NRMC  Oakland 
performed  approximately  2.6  million  tests  (including  quality 
controls) .  The  clinical  laboratory  at  Wright  Patterson  is  smaller 
than  that  at  Oakland,  with  an  annual  workload  of  1.6  million  tests  per 
year,  while  that  at  Eisenhower  is  larger,  with  a  volume  of  4.6  million 
tests  per  year. 
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TABLE  1 


CLINICAL  LABORATORY  AND  HOSPITAL  SUMMARY  STATISTICS 


NRMC 

Oakland 

Wright 

Patterson 

Eisenhower 

Number  of  Staff 

90 

42 

112 

Annual  Workload  (millions  of  tests) 

2.6 

1.6 

4.6 

Annual  Workload  (millions  of  CAP  units) 

8.9 

4.8 

15.2 

Hospital  Admissions 

14,600 

8,335 

13,096 

Hospital  Patient  Days 

89,000 

104,000 

118,110 

Hospital  Outpatient  Visits 

376,000 

400, 000 

774,000 

Average  Inpatient  Census 

244 

285 

324 
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The  chree  sites  differed  in  number  and  configuration  of  terminals 
supported  by  the  TRILAB  system  (Table  2) .  Oakland  obtained  42 
terminals,  of  which  18  were  located  in  provider  areas  (inpatient  units 
and  outpatient  clinics).  Wright  Patterson  received  25  terminals,  of 
which  14  were  located  in  provider  areas.  Eisenhower  received  36 
terminals,  of  which  7  were  located  in  provider  areas. 

TABLE  2 

CRT  CONFIGURATION 


Laboratory 
Inpatient  Units 
Outpatient  Areas 
Other 


NRMC  OAKLAND 
20 
12 
6 
_4 

Total  42 


Wright  Patterson 
9 
6 
8 

_2 

25 


Eisenhower 

29 

6 

1 


36 


The  sites  differed  also  in  terms  of  utilization  of  the  capability 
of  the  system  to  have  instruments  interfaced  on-line.  (This  permits 
on-line  inputting  of  test  results  directly  into  the  computer,  rather 
than  manually  inputting  results  through  terminals.)  Oakland  had  the 
SMAC  (high-volume  chemistry  analyzer) ,  Coulter  S+  (high  volume 
Instrument  used  in  hematology)  and  Clintech  instruments  on-line. 
Wright  Patterson  had  only  the  Hycel  (high-volume  chemistry  analyzer) 
on-line.  Eisenhower  had  the  SMAC,  ABA  200,  Clintech,  Beckman  Astron, 
and  Coulter  S+  instruments  on-line. 

Major  differences  between  the  baseline  (manual)  system  and  the 
post-implementation  (TRILAB)  operations  of  laboratory  services  were: 

•  Providers  in  locations  provided  with  terminals  received 
results  via  terminal  as  soon  as  they  were  available, 
instead  of  having  to  wait  for  telephone  calls  (in  the 
case  of  STAT  and  urgent  requests)  or  the  completed  test 
request  slip  (in  the  case  of  routine  requests). 
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•  The  reports  received  by  providers  were  cumulative, 
formatted  and  on  full-size  paper,  instead  of  on  the 
original  request  slips.  Medical  Records  received  a 
single  cumulative  discharge  report  for  inpatients,  rather 
than  a  number  of  slips  for  each  day's  tests. 

•  Within  the  laboratory,  work  sheets  were  prepared  by  the 
computer  Instead  of  being  prepared  manually. 

•  For  those  instruments  which  were  not  on-line  to  the 
computer,  technicians  entered  results  into  the  computer 
memory  via  terminal,  instead  of  writing  the  results  on 
the  original  test  request  slips.  For  those  Instruments 
which  were  on-line,  the  results  were  automatically 
entered  into  the  computer  memory.  Review  of  test  results 
was  expedited  in  that  normal  values  and  outliers  were 
automatically  identified,  as  well  as  unusual  changes  from 
previous  patients'  test  results. 

•  The  computer  produced  a  number  of  management  reports  for 
the  laboratory  which  previously  were  prepared  manually. 

•  Test  status  and  results  could  be  obtained  via  inquiry 
from  any  terminal  connected  to  the  system,  instead  of  by 
looking  up  records  or  telephoning  the  laboratory. 

C.  EVALUATION  APPROACH 

The  evaluation  plan  for  the  TRILAB  system,  developed  by  Analytic 
Services,  Inc.  (ANSER) ,  described  36  hypotheses  regarding  the  poten¬ 
tial  impact  of  TRILAB  on  the  clinical  laboratory  and  MTF.  These  were 
grouped  into  the  following  four  areas: 

•  personnel  time, 

•  satisfaction  and  perception, 

•  information  attributes,  and 

•  cost. 

evaluation  called  for  a  before-and-af ter  study  comparing 
m  oratory  operation  with  operation  of  the  laboratory  using  the 

i  :er  system.  Two  periods  of  data  collection  were  planned: 

the  ,a  under  manual  operation  before  the  Installation  of  TRILAB, 

and  the  Y  Period  when  TRILAB  became  operational. 
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This  before-and-after  approach  was  followed  in  the  plans  for  the 
three  TRILAB  evaluations.  The  elements  to  be  measured,  the  data 
collection  techniques,  and  the  data  obtained  at  each  site  during  the 
baseline  surveys,  are  summarized  in  Exhibit  1. 

Four  types  of  data  were  collected  by  ANSER  during  the  baseline 
study  at  NRMC  Oakland: 

•  Time  spent  by  personnel  within  the  laboratory  in 
information  handling  activities  (using  work  sampling  and 
timed  observations) ; 

•  Performance  of  services  (turnaround  time  for  test  results 
in  the  laboratory  process  time;  transcription 
discrepancies;  number  of  telephone  inquiries  about  test 
results;  and  patient  waiting  time); 

•  Staff  perceptions  of  performance  of  services  (staff 
questionnaire  survey) ;  and 

•  Staff  and  patient  satisfaction  (staff  and  patient 
questionnaire  survey) . 

AT  DDEAMC  two  types  of  quantitative  information  were  obtained  by 

(2) 

Arthur  D.  Little,  Inc.  on  performance  of  services: 

•  turnaround  time  for  laboratory  tests,  and 

•  number  of  inquiries  to  the  laboratory  about  test  results. 

The  surveys  carried  out  at  Eisenhower  and  Wright  Patterson  were 
similar  to  those  carried  out  at  NRMC  Oakland,  with  some  modifications 
made  to  the  survey  instruments,  e.g.,  to  reflect  the  fact  that  Wright 
Patterson  previously  had  had  the  Air  Force  Clinical  Automated  System 
(AFCLAS) . 

The  post-implementation  evaluation  plan  was  developed  with  the 
following  considerations: 

(1)  In  order  to  utilize  the  baseline  data  to  the  maximum 
extent,  and  to  make  the  before-and-after  comparison  as 
consistent  and  meaningful  as  possible,  the  same 
evaluative  measures,  data  collection  methodologies,  and 
data  collection  instruments  used  in  the  baseline  period 
were  used  in  the  post-implementation  study  to  the 
extent  possible.  It  was  necessary,  however,  to  modify 
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Patient  Satisfaction  Patient  satisfaction  with  Self-adoiinis-  Satisfaction  ratings  Satisfaction  ratings  Satisfaction  ratings 

with  Laboratory  Services  laboratory  services  tered  question-  using  Likert  scale  using  Likert  scale  using  Likert  scale 


Che  data  collection  methodology  and  instruments  in 
several  instances,  in  view  of  the  differences  in 
procedures  in  the  laboratory  because  of  the  TRILAB 

system. 

(2)  Samples  sizes  were  chosen  to  provide  as  reliable  as 

possible  data,  given  the  data  available  from  the 

(2) 

baseline  evaluation. 

(3)  Where  the  baseline  data  were  unavailable  or 

insufficient  for  evaluation  purposes,  an  effort  was 

made  to  collect  such  information  via  interviews  and 
review  of  any  available  reports. 

The  post-implementation  data  collection  plan  was  organized  under 
the  following  data  collection  methodologies: 

(1)  Work  sampling  at  NRMC  Oakland:  to  collect  data  on 
distribution  of  laboratory  personnel  activities  in  the 
post-implementation  for  comparison  with  distribution  of 
activities  in  the  baseline.  The  major  objective  was  to 
determine  whether  time  spent  in  information  handling 
activities  had  changed. 

(2)  Process  time  study  at  NRMC  Oakland:  to  determine 
whether  process  times  for  return  of  test  results  had 
changed . 

(3)  Volume  of  telephone  calls  to  the  laboratory  at  NRMC 
Oakland:  to  determine  whether  changes  had  occurred  in 
the  volume  of  telephone  inquiries  to  the  laboratory  as 
a  result  of  the  availability  of  results  via  the  TRILAB 
system. 

(A)  Surveys  of  laboratory  staff,  providers  and  patients  at 
NRMC  Oakland  and  Wright  Patterson:  to  determine 
whether  satisfaction  with  clinical  laboratory  services 
had  changed. 

(5)  Interviews  and  supplementary  data  collection:  to 

obtain  cost  data,  volume  data,  and  other  information. 

Exhibit  2  summarizes  the  relationship  between  the  evaluation 
elements  and  the  data  obtained  in  the  post-implementation  study. 
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III.  EVALUATION  RESULTS 


This  chapter  summarizes  the  evaluation  data  collected  during  the 
baseline  and  post-implementation  evaluations.  Further  details  on 
evaluation  findings  are  presented  in  the  remaining  volumes  of  this 
report.  This  chapter  is  organized  according  to  the  system  goals  and 
evaluation  measures:  (a)  personnel  time  devoted  to  information 
handling  (obtained  only  at  NRMC  Oakland) ;  (b)  performance  of  service 
measures  (also  obtained  only  at  NRMC  Oakland);  (c)  attitudes  and 
perceptions  about  services  (collected  via  survey  questionnaires  at 
NRMC  Oakland  and  Wright  Patterson);  and  (d)  interview  information 
(collected  at  all  three  sites). 

A.  PERSONNEL  TIME  DEVOTED  TO  INFORMATION  HANDLING 

One  goal  of  the  TRILAB  system  was  to  reduce  time  spent  by 
laboratory  staff  in  clinical  or  information-handling  activities.  In 
both  the  baseline  and  post-implementation  studies  at  NRMC  Oakland, 
time  spent  by  laboratory  personnel  in  information  handling  activities 
was  measured  by  an  extensive  work  sampling  program  conducted  in  the 
three  major  laboratory  sections:  Chemistry,  Hematology,  and 
Bacteriology. 

1.  Comparison  of  Baseline  and  Post-Implementation  Results 

Table  3  compares  the  time  devoted  to  information  handling 
activities  in  the  baseline  and  post-implementation  periods,  in  terms 
of  both  percent  of  time  and  estimated  hours  per  week.  The  comparisons 
of  percentage  of  time  are  considered  more  accurate  than  the 
comparisons  of  estimated  hours  per  week,  due  to  difficulties  in 
interpretation  of  the  estimates  of  weekly  hours  and  staffing  levels  in 
the  three  sections  during  the  baseline  sampling  period.  (Baseline 
hours  per  week  for  Chemistry  have  been  adjusted  for  comparison 
purposes  to  reflect  the  fact  that  the  Nuclear  Medicine  section  was  not 
included  in  post-implementation  sampling;  Chemistry  staff  hours  were 
reduced  by  12.5  percent — the  percent  accounted  for  by  Nuclear  Medicine 
staff —  to  make  the  results  comparable.) 
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TABLE  3 


COMPARISON  OF  BASELINE  AND  POST- IMPLEMENTATION  INFORMATION  HANDLING  TIMES 
TRILAB  EVALUATION  IN  THE  CLINICAL  LABORATORY  AT  NRMC  OAKLAND 


Percent  of  Time 


Base¬ 

line 

Post- 

Implementation 

Chemistry 

37.3% 

38.5% 

Hematology 

18.1 

15.4 

Microbiology 

22.6 

21.0 

All 

27. 2a 

24.6 

Hours  per  Week 


Differ¬ 

ence 

Base¬ 

line 

Post- 

Implementation 

Differ¬ 

ence 

1.2  % 

208. 6a 

161.7 

-46.9 

-2.7b 

68.0 

73.9 

5.9 

-1.  6b 

103.5 

84.0 

-19.5 

-2.  6b 

380.1 

319.6 

-60.5 

I 


r 
L 

r 


D 

D 


D 


y 


g 

Adjusted  for  Nuclear  Medicine  staff. 

Difference  is  statistically  significant  at  95$  confidence  level. 
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Overall,  the  percentage  of  time  devoted  to  information  handling 
activities  was  2.6  percent  lower  in  the  post-implementation  than  in 
the  baseline  period;  this  difference  was  statistically  significant. 

Statistically  significant  (at  a  95  percent  confidence  level) 
reductions  in  the  percentage  of  time  devoted  to  information  handling 
activities  were  observed  in  Hematology  and  Microbiology  sections  (2.7 
percent  and  1.6  percent,  respectively).  An  increase  of  1.2  percent  of 
time  devoted  to  information  handling  activities  was  observed  for 
Chemistry.  This  increase,  however,  was  not  statistically  significant. 
The  overall  reduction  in  time  devoted  to  information  handling  between 
the  two  study  periods  was  about  60  hours  per  week.  Part  of  this 
reduction,  however,  was  due  to  the  difference  in  staffing.  At  the 
staffing  level  during  the  post-implementation  study,  a  reduction  of 
2.6  percent  was  equivalent  to  a  reduction  of  35.8  hours  per  week 
devoted  to  information  handling  activities,  or  slightly  less  than  one 
FTE. 

Table  4  compares  the  percentage  of  time  devoted  to  several 
selected  activities  in  the  two  study  periods.  Overall,  time  devoted 
to  processing  of  test  results  increased  .y  approximately  1.2  percent. 
As  might  be  expected,  time  devoted  to  transcription  and  recording  of 
test  ’-esults  was  reduced  by  about  0.9  percent,  equivalent  to  about  16 
hours  per  week.  Time  devoted  to  compilation  of  workload  statistics, 
which  accounted  for  1.5  percent,  or  21  hours  per  week  of  time  in  the 
baseline  period,  was  eliminated  in  the  post-implementation  period 
because  the  computer  system  assumed  this  function.  Time  devoted  to 
quality  control  logging,  calculation,  and  updating  was  reduced  from 
2.9  percent  to  0.7  percent  of  total  time,  or  by  31  hours  per  week. 

Time  spent  away  from  the  area  was  25.2  percent  in  the 
post-implementation  period  compared  with  32.7  percent  in  the  baseline. 
It  is  not  clear  whether  this  was  due  to  differences  in  sampling 
methodology,  or  whether  the  staff  did  spend  more  time  in  the 
laboratory  sections,  possibly  as  a  result  of  reduction  in  available 
staff. 
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2.  Conclusions 


It  was  concluded  that  time  devoted  to  information  handling 
activities  in  the  post-implementation  period  was  approximately  2.6 
percent  lower  than  in  the  baseline.  Based  on  current  staffing,  this 
was  equivalent  to  a  net  reduction  of  34  hours  per  week,  (day  shift, 
Monday  to  Friday)  devoted  to  information  handling  activities  in  the 
Chemistry,  Hematology  and  Microbiology  sections.  As  expected,  there 
were  reductions  in  time  devoted  to  transcription  and  recording  of  test 
results,  compilation  of  workload  statistics,  and  for  quality  control 
reporting. 

B.  PERFORMANCE  OF  SERVICES 

Another  goal  of  the  TRILAB  system  was  to  make  information 
available  to  providers  with  increased  efficiency.  Data  on  two  types 
of  performance  measures  were  collected  at  NRMC  Oakland  during  the 
baseline  and  post-implementation  periods: 

•  turnaround  time  for  laboratory  requisitions;  and 

9  volume  of  telephone  calls  inquiring  about  test  results. 

1.  Turnaround  Times 

a.  Comparison  of  Baseline  and  Post-Implementation  Laboratory 

Process  Times 

Table  5  compares  process  times  observed  in  the  two  study  periods. 
In  the  case  of  the  STAT/urgent  tests,  the  post-implementation  averages 
presented  are  for  the  "CRT  process  time"  (time  results  available  via 
the  CRT) ,  as  being  most  comparable  to  the  process  time  measured  in  the 
baseline  period  (time  when  results  were  telephoned  back  to  the 
requesting  units).  In  the  case  of  routine  tests,  the  final  results 
available  via  CRT  are  presented,  as  being  most  comparable  to  process 
times  measured  in  the  baseline  period  (completed  results  slips 
available  for  pickup) . 

The  results  suggest  that  process  times  were  reduced  in  the 
post-implementation  period  compared  to  the  baseline  period  for 
Hematology  STAT/urgent  tests  (by  0.4  hours)  and  for  Hematology  routine 
tests  (by  2.6  hours);  these  differences  were  statistically  significant 
at  the  95  percent  confidence  level.  The  process  times  for  STAT/urgent 
and  routine  Chemistry  tests  increased  (by  0.23  hours  apd  8.3  hours. 
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TABLE  5 

COMPARISON  OF  3ASELINE  AND  POST- IMPLEMENTATION  AVERAGE  PROCESS  TIMES 
TRILAB  EVALUATION  IN  THE  CLINICAL  LABORATORY  AT  NRMC  OAKLAND 


Mean  Turnaround  Time  (hours) 


STAT/Urgent 

Baseline 

Post- Implementation 

Difference 

F  Test 

Chemistry 

1.  27a 

1.50b 

0.23 

2.6" 

Hematology 

1.253 

.  85b 

-0.40* 

1.07 

Routine 

Chemist  ry 

16. 1C 

9/  -d 

.  <4 

8.  3 

2 .  S* 

Hematology 

4.7C 

2.1d 

-2.6* 

12.  j* 

Microbiology 

39. 1C 

59. 6d 

20.5* 

3.5* 

^Time  from  receipt  of  specimen  by  laboratory  to  telephoning  resulcs. 

time  from  accessioning  to  availability  of  first  results  via  terminal. 

^Time  from  receipt  of  specimen  to  availability  of  results  at  distribution  box. 

Time  from  accessioning  to  availability  of  final  results  via  terminal. 
*Statistically  significant  difference  at  95%  confidence  level. 
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respectively);  these  differences,  however,  were  not  statistically 
significant.  Average  process  times  for  Microbiology  tests  increased 
by  about  20  hours;  this  difference  was  statistically  significant. 

The  differences  observed  must  be  interpreted  with  caution  for  the 
following  reasons: 

•  In  the  case  of  routine  tests,  the  process  times  are  not 
entirely  comparable  because  in  the  baseline  period  the 
process  time  represents  the  time  when  the  requisition 

slip  was  available  for  pickup,  where  the  post-implementation 
process  time  represents  the  time  that  the  result  was  in 
fact  available  to  the  requester  (via  terminal  look-up) . 

The  time  between  availability  of  the  completed  requisition 
slip  and  pickup  or  availability  of  the  result  to  the 
requester  was  not  measured  in  the  baseline  study  at 

k 

NRMC  Oakland.  At  DDEAMC,  however,  total  turnaround 
time  was  measured  in  addition  to  "process  time;"  the 
difference  varied  between  17  hours  and  20  hours,  on 
average.  Thus  routine  results  were  available  to  the 
provider  (via  terminal  inquiry)  considerably  sooner  in 
the  post-implementation  period  for  routine  Chemistry 
and  Hematology  tests,  and  probably  in  about  the  same 
time  for  Microbiology  tests. 

•  During  the  post-implementation  period,  the  tests  to  be 
sampled  in  each  test  type  category  were  chosen  by  using 
an  appropriate  "skip"  interval  between  tests,  in  order 
to  obtain  a  random  sample  representative  of  turnaround 
times.  It  is  not  clear  how  the  sample  for  the  baseline 
period  data  collection  was  chosen;  as  indicated  in  the 
baseline  report,  much  of  the  data  was  unusable  due  to 
lack  of  identification  of  data  sheets,  obvious  errors 
(e.g.,  receipt  dates  being  later  than  dates  of  results 
report)  and  errors  in  the  calculation  of  process  times, 

*A  mini  baseline  evaluation  was  conducted  to  evaluate  TRILAB  at 

Eisenhower  AMC.  Complete  results  are  summarized  in  the  baseline 

(2) 

report.  Volume  III 
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resulting  in  some  questionable  data.  In  fact,  there  is 
evidence  that  the  distributions  (spreads)  of  test  result 
process  times  are  not  the  same  in  the  two  periods,  as 
measured  by  the  statistical  F  Test.  As  indicated  in 
Table  6,  the  distribution  of  test  results  in  the  two 
study  periods  are  significantly  different  for  each  test 
category,  except  for  Hematology  STAT/urgent  tests. 

•  Operating  conditions  in  the  laboratory  were  not  the  same 
in  the  two  study  periods.  The  Hematology  Laboratory  had 
obtained  a  new  Coulter  S+  instrument,  which  had  a  higher 
throughput  rate  than  the  previous  instrument  (the  Coulter 
was  the  major  instrument  utilized  in  the  Hematology  section) . 
Staffing  in  the  Chemistry  section,  as  mentioned  earlier,  was 
somewhat  reduced  from  that  of  the  baseline  period.  Also, 
staff  were  preparing  for  the  Joint  Accreditation  visit, 
which  may  have  resulted  in  fewer  staff  being  available  for 
production  of  tests. 

•  The  apparent  increase  in  Microbiology  test  times  is 
difficult  to  explain,  except  for  the  possibility  that 
the  small  baseline  sample  (16  observations)  may  not  be 
representative  of  the  tests  carried  out  in  the  Microbiology 
section  (specific  tests  and  requesting  locations  were  not 
provided  in  the  baseline  data);  as  mentioned  above,  the 
distributions  of  process  times  were  significantly  different. 
Since  test  times  in  Microbiology  are  longer  and  vary  so 
much,  depending  on  type  of  specimen  and  results  (positive 

or  negative) ,  differences  in  the  type  of  tests  sampled  in 
the  baseline  and  post-implementation  periods  could  result 
in  considerably  different  test  times,  which  masked  any 
differences  in  reporting  times. 

b.  Conclusions 

Given  the  above  qualifications,  the  following  conclusions  may  be 
drawn: 
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(1)  For  STAT/urgent  tests,  process  times — times  between 
receipt  of  requisition  and  transmission  of  test  results 
to  provider  locations  (via  telephone  in  the  baseline 
period  and  via  terminal  in  the  post-implementation 
period) — were  either  unchanged  (Chemistry)  or  reduced 
(Hematology) . 

(2)  For  routine  tests,  results  were  available  to  provider 
locations  (via  terminal)  sooner  in  the  post-implementation 
period  than  in  the  baseline  period  for  Chemistry  and 
Hematology,  and  in  about  the  same  time  period  or  sooner 
for  Microbiology.  Hard-copy  daily  reports  were  generally 
available  to  providers  later  than  the  completed  results 
requisition  slips  were  in  the  baseline  period.  Interim 
hard-copy  reports  (for  the  surgical  floors)  were  available 
sooner  in  the  post-implementation  period  for  Hematology 
tests,  and  in  approximately  the  same  time  for  Chemistry 
and  Microbiology  tests. 

2.  Number  of  Telephone  Calls 

a.  Comparison  of  Baseline  and  Post-Implementation  Results 

Table  6  provides  a  comparison  of  the  number  of  calls  per  day 
received  by  the  laboratory  at  NRMC  during  the  two  study  periods.  In 
the  post-implementation  period  study,  the  laboratory  received  an 
average  of  67.4  calls  per  day.  The  total  number  of  calls  received  per 
day  during  the  post-implementation  period  was,  therefore,  two-thirds 
that  received  during  the  baseline  period.  The  distribution  of  calls 
by  type  was  similar  to  that  received  during  the  baseline  period,  with 
the  majority  of  calls  (51  percent)  requesting  information  from  the 
laboratory  with  regard  to  test  results.  The  number  of  calls  received 
was  lower  in  each  category  of  call  request,  except  for  calls  to 
technicians,  which  doubled  from  9  calls  per  day  during  the  baseline 
period  to  about  18  calls  per  day  during  the  post-implementation 
period.  The  reason  for  the  increase  in  this  category  is  not  known. 

Based  on  the  170,150  patient  tests  reported  performed  during 
October  1982,  the  average  daily  test  load  was  5,489  tesxs  per  day 
for  the  31-day  period.  Assuming  the  number  of  telephone  call^  to  the 
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TABLE  6 

COMPARISON  OF  BASELINE  AND  POST- IMPLEMENTATION  PERIOD  TELEPHONE  CALL  RESULTS 
TRILAB  EVALUATION  IN  THE  CLINICAL  LABORATORY  AT  NRMC  OAKLAND 


Number  per  Eight-Hour  Da> 


Ratio  of  3aseline  to 
Post- Implementation 


Type  of  Call 

Baseline 

Post- Implementation 

Unnormalized 

Normalized 
for  Workload 

For  filed  results 

12.5 

2.9 

0.23 

0.20 

Information  from  laboratory 

62.4 

34.4 

0.55 

0.48 

Supervisor 

12.9 

3.8 

j.  68 

0.59 

Technician 

9.0 

17.8 

1.98 

1.  72 

General  Information 

5.6 

3.5 

0.63 

0.55 

TOTAL 

102.4 

67.4 

0.66 

C  •  Zi  i 
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laboratory  was  similar  on  weekdays  and  weekends,  this  represented  an 
average  of  one  call  for  every  81.4  tests  performed.  This  was  a 
reduction  of  43  percent  from  the  average  of  1  call  for  every  46.5 
tests  measured  during  the  baseline  period. 

It  should  be  noted  that  not  all  nursing  stations  or  clinics  had 
terminals  and  not  all  areas  of  the  laboratory  (viz.,  Nuclear  Medicine, 
Pathology  and  Blood  Bank)  were  on  the  system.  If  additional  terminals 
are  obtained,  the  volume  of  telephone  calls  could  be  expected  to  be 
further  reduced. 

c.  Conclusions 

The  volume  of  telephone  calls  to  the  laboratory  was  considerably 
reduced  by  implementation  of  the  TRILAB  system.  Normalized  to  study 
period  workloads,  the  volume  of  telephone  calls  in  the  post¬ 
implementation  period  was  almost  half  (57  percent)  that  in  the  base¬ 
line  study  period. 

C.  STAFF  PERCEPTIONS  OF  LABORATORY  SERVICES 

Survey  questionnaires  were  distributed  to  medical  staff,  nurses, 
administrative  corpsmen  and  clerical  staff,  and  laboratory  staff 
during  the  baseline  and  post-implementation  studies.  The  purpose  of 
the  questionnaires  was  to  determine  the  degree  of  satisfaction  of 
various  providers  and  staff  with  regard  to  their  perceptions  of 
laboratory  operations.  The  baseline  survey  at  Oakland  was  designed 
and  carried  out  by  ANSER,  while  the  surveys  at  Wright  Patterson  were 
designed  and  carried  out  by  A.  D.  Little,  Inc.  The  results  for  Wright 
Patterson  are  somewhat  more  complete. 

Questions  were  included  with  regard  to  perceptions  about  such 

factors  as: 

•  relations  with  laboratory  personnel; 

•  legibility,  quality,  accuracy  and  format  of 
laboratory  reports; 

•  amount  of  time  required  to  obtain  test  results; 

•  promptness  and  completeness  of  laboratory  reports 
in  patient  records;  and 

•  ease  of  and  amount  of  time  required  to  obtain 
information  by  telephone. 
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BASELINE  AND  POST- IMPLEMENTATION  SATISFACTION  LEVELS  OF  USERS  OF  CLINICAL  LABORATORY  SERVICES 
REGARDING  PERFORMANCE  OF  THE  CLINICAL  LABORATORY  AT  NRMC  OAKLAND 
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BASELINE  AND  POST- IMPLEMENTATION  SATISFACTION  LEVELS  OF  USERS  OF  CLINICAL  LABORATORY  SERVICES 
REGARDING  PERFORMANCE  OF  THE  CLINICAL  LABORATORY  AT  WRIGHT  PATTERSON  MEDICAL  CENTER  AND  REGIONAL  HOSPITAL' 
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Baseline,  46  respondents;  Post -implementation,  7b  respondents:  30  physicians,  31  nurse/p. a.,  12  administrators,  3  others. 


1 


Respondents  were  asked  to  rate  satisfaction  with  each  factor. 
These  ratings,  assigned  scale  values  of  5,  4,  3,  2,  1  (the 

conventional  Likert  scale)  were: 

5  highly  satisfactory; 

4  satisfactory; 

3  neither  satisfactory  nor  unsatisfactory; 

2  unsatisfactory;  or 

1  highly  unsatisfactory. 

1 .  User  Attitudes 

a.  Clinical  Laboratory  Performance 

Tables  7  and  8  summarize  the  changes  in  mean  satisfaction  levels 
at  NRMC  Oakland  and  Wright  Patterson  regarding  clinical  laboratory 
services. 

At  Oakland,  satisfaction  levels  of  physicians,  nurses  and 
physicians'  assistants,  and  other  users  increased  in  all  categories, 
with  one  exception  (satisfaction  by  "other  [administrative]  users" 
with  regard  to  overall  laboratory  performance) .  Physicians  showed  the 
greatest  increase  in  satisfaction  levels  with  regard  to  turnaround 
times  of  both  routine  and  STAT  tests.  Nurses  and  physicians' 
assistants  showed  the  greatest  increase  in  satisfaction  with  regard  to 
accuracy  of  results,  completeness  of  laboratory  reports,  and  test 
turnaround  times. 

The  survey  results  at  Wright  Patterson  show  that  TRILAB  have 
resulted  in  positive  changes  in  physician  attitudes  towards  laboratory 
services,  as  evidenced  in  Table  8.  Satisfaction  with  all  aspects  of 
performance  of  laboratory  services  increased  from  the  baseline  to  the 
post-implementation  survey.  Physician  opinion  in  the  baseline  was 
largely  divided,  as  indicated  by  weighted  means  of  approximately  3.0. 
The  post-implementation  period  survey  results  indicate  that,  for  the 
most  part,  physicians  were  basically  "satisfied"  with  laboratory 
services.  The  most  marked  change  was  in  satisfaction  with  accuracy  of 
laboratory  results;  respective  weighted  mean  changes  from  the  baseline 
to  post-implementation  were  3.0  to  4.1,  or  "undecided"  to  "somewhat 
satisfied."  There  was  also  a  significant  increase  in  satisfaction 
with  routine  results  turnaround  time,  from  2.8  to  3.7. 
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The  surveys  at  both  NRMC  Oakland  and  Wright  Patterson  thus  show 
positive  changes  in  user  attitudes  towards  laboratory  services. 

b.  Frequency  of  Problem  Occurrences 

Users  at  Oakland  NRMC  were  asked  to  compare  relative  frequency  of 
problem  events  under  TRILAB  operations  with  previous  laboratory 
(manual)  operations.  Only  those  users  who  were  at  the  facility  before 
TRILAB  was  installed  (in  February  1982)  were  asked  to  answer  this 
question.  Two-thirds  of  the  respondents  to  the  post- implementation 
survey  fell  into  this  category. 

The  TRILAB  system  received  relatively  "high  marks"  for  most 
problem  categories  (Table  9) .  The  median  response  was  that  the 
following  occurred  "less  frequently"  with  the  TRILAB  system: 

•  tests  repeated  due  to  delays  (43.4  percent  felt  they 
occurred  less  frequently); 

•  tests  repeated  due  to  lost  results  (46.7  percent);  and 

•  telephone  calls  to  the  laboratory  (64.6  percent). 

The  median  response  to  "tests  repeated  due  to  inaccurate  results" 
was  that  this  occurred  with  similar  frequency  (40.3  percent). 
Unnecessary  duplication  of  report  data  also  received  a  median  response 
of  "similar  frequency  with  TRILAB." 

Table  10  summarizes  the  responses  received  at  Wright  Patterson  to 
the  same  questions.  Again,  the  median  response  was  that  the  following 
problems  occurred  "less  frequently"  with  the  TRILAB  system: 

•  tests  repeated  due  to  delays  (58.2%  felt  they  occurred 
less  frequently) ; 

•  tests  repeated  due  to  lost  results  (63.6%); 

•  telephone  calls  to  the  laboratory  (65.5%);  and 

•  unnecessary  duplication  or  report  data  (54.7%). 

"Tests  repeated  due  to  inaccurate  results"  were  judged  to  occur  with 
similar  frequency  (38.9%)  or  less  frequently  (44.4%). 

Thus,  the  questionnaire  surveys  at  both  NRMC  Oakland  and  at 
Wright  Patterson  indicated  that  users  felt  that  problem  occurrences 
were  less  frequent  after  implementation  of  TRILAB. 
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TABLE  9 


COMPARISON  OF  TRILAB  WITH  MANUAL  OPERATIONS 
POST-IMPLEMENTATION  PERIOD  OF  USERS 
NRMC  OAKLAND* 


Change  in  Frequency  (Percent  of  Respondents) 


More  Frequently 
With  TRILAB 

Similar 

Frequency 

Less 

Frequently 

Never 

with  TRILAB 

No 

Opinion 

Tests  repeated  due 
to  delays 

9.3% 

23.1% 

43.4% 

1.6% 

22.5% 

Tests  repeated  due 
to  lost  results 

6.0 

24.7 

46.7 

2.7 

19.8 

Tests  repeated  due 

to  inaccurate  results 

0.6 

40.3 

29.3 

2.2 

27.6 

Telephone  calls  to 
Laboratory 

6.1 

14.4 

64.6 

2.2 

12.7 

Unnecessary  duplication 
of  report  data 

35.7 

13.2 

24.2 

2.2 

24.7 

*Completed  by  users  who  were  also  at  the  hospital  prior  to  installation  of  TRILAB. 
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TABLE  10 


COMPARISON  OF  TRILAB  WITH  MANUAL  OPERATIONS 
POST-IMPLEMENTATION  PERIOD  OF  USERS 
WRIGHT  PATTERSON  MEDICAL  CENTER* 


Change 

in  Frequency  (Percent  of  Respondents) 

More  Frequently 
With  TRILAB 

Similar 

Frequency 

Less  Never 

Frequently  with  TRILAB 

No 

Opinion 

Tests  repeated  due 
to  delays 

3.6% 

21.8% 

58.2% 

3.6% 

12.7% 

Tests  repeated  due 
to  lost  results 

3.6 

23.6 

63.6 

3.6 

5.5 

Tests  repeated  due 

to  inaccurate  results 

- 

38.9 

44.4 

3.7 

13.0 

Telephone  calls  to 
Laboratory 

3.6 

14.5 

65.5 

14.5 

1.8 

Unnecessary  duplication 
of  report  data 

9.4 

17.0 

54.7 

9.4 

9.4 

*Completed  by  users  who  were  also  at  the  hospital  prior  to  installation  of  TRILAB. 
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2.  Attitudes  of  Clinical  Laboratory  Personnel 

Separate  baseline  and  post-implementation  survey  questionnaires 
were  administered  to  the  personnel  on  the  laboratory  staff  of  NRMC 
Oakland  and  Wright  Patterson.  Detailed  results  are  presented  in 
Volumes  II  and  III  of  this  report. 

a.  Relative  Frequency  of  Problem  Occurrences 

When  asked  to  compare  operations  under  TRILAB  with  manual 
operations  prior  to  installation  of  TRILAB  (Table  11),  respondents  at 
NRMC  Oakland  (who  had  been  with  the  laboratory  prior  to  installation 
of  TRILAB)  indicated  that  the  following  occurred  less  frequently  with 
TRILAB: 

•  telephone  calls  to  providers  (73  percent); 

•  time  spent  on  manual  record-keeping  (47  percent);  and 

•  duplication  of  information  (41  percent). 

The  following  were  judged  to  occur  with  similar  frequency  under 
TRILAB: 

•  repeating  tests  due  to  inaccurate  results  (53  percent); 
and 

•  transcription  discrepancies  (43  percent). 

Laboratory  personnel  at  Wright  Patterson  were,  overall,  in 
agreement  that  frequency  of  common  laboratory  discrepancies  either 
decreased  or  occurred  with  similar  frequency  in  the  post¬ 
implementation  period  (Table  12) .  The  following  were  judged  to  occur 
less  frequently: 

t  telephone  calls  to  providers  (82%) ;  and 

•  repeating  tests  due  to  inaccurate  results  (36%) . 

The  following  were  estimated  as  occurring  with  similar  frequency: 

•  duplication  of  information; 

•  time  spent  on  manual  record  keeping;  and 

•  transcription  discrepancies  (55%) . 

Thus  laboratory  personnel  at  both  NRMC  Oakland  and  Wright 
Patterson  indicated  that  frequency  of  events  which  interfered  with 
flow  of  work,  in  particular  telephone  calls,  had  decreased. 
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TABLE  11 


COMPARISON  OF  TRILAB  WITH  MANUAL  OPERATIONS 
POST-IMPLEMENTATION  PERIOD  OF  LABORATORY  PERSONNEL 

NRMC  OAKLAND 


Change  in  Frequency  (Percent  of  Respondents) 


More  Frequently 

Similar 

Less 

Never 

No 

With  TRILAB 

Frequency 

Frequently 

with  TRILAB 

Opinion 

Telephone  calls  to 
inpatient  units/ 
outpatient  clinics 

6.7% 

20.0% 

73.3% 

0% 

0 

Duplication  of 
information 

20.7 

17.2 

41 . 4 

10.3 

10.3 

Necessity  of  repeating 
tests  due  to 
inaccurate  results 

3.3 

53.3 

30.0 

13.3 

10.0 

Time  spent  on  manual 
record  keeping 

20.0 

33.3 

46.7 

0 

0 

Discrepancies  in 
transcription 

23.3 

43.3 

20.0 

3.3 

10.0 

/ti  Arthur  D.  Little,  Inc. 


29 


TABLE  12 


COMPARISON  OF  TRILAB  WITH  MANUAL  OPERATIONS 
POST-IMPLEMENTATION  PERIOD  OF  LABORATORY  PERSONNEL 
WRIGHT  PATTERSON  MEDICAL  CENTER 


_ Change  In  Frequency  (Percent  of  Respondents) 

More  Frequently  Similar  Less  Never 

With  TRILAB  Frequency  Frequently  with  TRILAB 


Telephone  calls  to 
inpatient  units/ 

outpatient  clinics  0  % 

Duplication  of 

information  9.1 

Necessity  of  repeating 
tests  due  to 

inaccurate  results  9.1 

Time  spent  on  manual 

record  keeping  0 

Discrepancies  in 

transcription  0 


9.1 X  81.8%  0  % 

45.5  36.4  0 

18.2  36.4  18.2 

81.8  18.2  0 

54.5  45.5  0 
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No 

Opinion 

9.1% 

9.1 

is.: 

o 

0 


3.  Patient  Satisfaction 


A  sample  of  patients  at  NRMC  Oakland  and  Wright  Patterson  were 
asked  to  complete  a  brief  questionnaire  regarding  their  satisfaction 
with  laboratory  services.  The  laboratory  at  NRMC  Oakland  received 
comparatively  high  ratings  by  patients  in  both  study  periods.  The 
overall  satisfaction  rating  remained  generally  unchanged  (4.5),  as  did 
satisfaction  with  waiting  times  for  specimen  taking  (4.2).  There  was 
a  decrease  in  satisfaction  rating  of  waiting  time  required  to  register 
(from  4,5  to  4.1).  Patients  were  also  asked  to  indicate  whether  they 
had  to  have  tests  repeated  due  to  lost  results  or  whether  they 
experienced  delays  due  to  incomplete  test  request  forms.  Most 
patients  indicated  that  they  "never"  or  "rarely"  experienced  such 
problems.  Two  to  three  percent  indicated  that  they  occurred  "often." 
About  22  percent  of  patients  added  comments  to  their  questionnaire. 
The  majority  of  these  were  laudatory,  and  referred  to  the  "excellent 
service"  received  at  the  facility  or  that  they  had  experienced  no 
problems . 

No  patient  data  were  available  for  analysis  in  the  baseline 
period  at  Wright  Patterson.  In  the  post-implementation  period, 
patients  were  on  average  "somewhat  satisfied"  with  clinical  laboratory 
services  at  Wright  Patterson,  including  time  waiting  to  be  served  in 
the  laboratory.  Delayed  or  repeated  tests  were  considered  to  occur  on 
average  between  "rarely"  and  "never." 

D.  RESULTS  OF  INTERVIEWS 

During  the  study  periods  and  implementation  monitoring  visits, 
interviews  were  held  with  a  number  of  laboratory  supervisors  and  with 
users,  with  regard  to  benefits  achieved  with  the  TRILAB  system,  and 
any  problems  encountered.  The  following  summarizes  the  results  of 
these  interviews. 

1.  Laboratory  Staff 

a.  Benefits  Achieved  in  Laboratory 
•  Telephone  Calls 

Laboratory  staff  at  all  three  sites  felt  that  a  very 
significant  decrease  had  occurred  in  telephone  calls  to  „ 
the  laboratory,  which  had  interrupted  work  flow  and  taken 
up  staff  time. 
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At  Oakland,  supervisors  estimated  that  telephone 
calls  to  the  laboratory  had  been  reduced  by  about  50 
percent,  because  of  availability  of  results  and  test 
status  to  providers  via  terminal  inquiry.  (The  study  of 
telephone  call  volume  indicated  that  telephone  call 
volume  decreased  by  43  percent.)  Staff  at  Eisenhower 
estimated  that  volume  of  telephone  calls  was  reduced  by 
40-50  percent  (in  Chemistry  and  Hematology),  equivalent 
to  savings  of  about  6-1/2  hours  per  day  of  staff  time. 

•  Workload  Reporting.  Supervisors  at  the  three  sites  felt 
that  TRILAB  accomplished  management  and  workload 
reporting  tasks  more  efficiently  than  in  the  baseline. 
Supervisors  at  Oakland  estimated  that  in  the  three  major 
sections  (Chemistry,  Hematology,  and  Microbiology), 
approximately  8.5  hours  per  week  in  total  were  saved  by 
having  the  TRILAB  system  produce  the  monthly  workload 
reports;  these  reports  were  previously  prepared  manually. 
(This  compares  with  the  reduction  of  21  hours  estimated 
from  the  work  sampling  study  data.)  At  Eisenhower, 
supervisors  estimated  a  savings  of  ten  hours  per  week 
(in  Chemistry  and  Hematology). 

•  Quality  Control  Reports.  Supervisors  indicated  that  the 
quality  control  reports  of  the  system  (Levev-Jennings 
charts)  were  a  significant  benefit.  Supervisors  at 
Oakland  estimated  that  11  hours  per  week  were  saved  in 
preparing  the  quality  control  reports,  which  were  pro¬ 
duced  by  the  TRILAB  computer  system.  Staff  at  Eisenhower 
estimated  staff  savings  of  three  hours  per  week. 

•  Patient  Exception  Reports.  It  was  estimated  at  Oakland 
that  approximately  ten  hours  per  week  were  saved  in 
review  of  patient  exception  reports,  due  to  the 
highlighting  of  abnormal  results  by  the  computer  system. 

•  Logging  of  Specimens  and  Preparation  of  Work  Sheets. 
Staff  at  Eisenhower  estimated  that  reduction  of  time 
spent  in  these  activities  averaged  14  staff  hours  per 
day. 
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•  Duplicate  Tests.  It  was  anticipated  that  the  number  of 
duplicate  or  repeated  tests  may  be  reduced  with  the 
TRILAB  system'  Laboratory  staff  suggested  that  this 
could  occur  for  two  reasons:  (1)  providers  could  easily 
check  on  the  status  of  tests,  and  would  be  less  inclined 
to  repeat  an  order  if  they  saw  a  test  was  pending  or  in 
process;  (2)  abnormal  results  and  unusual  "delta  checks" 
(abrupt  changes  from  the  previous  day's  results)  showed 
up  on  the  screen  as  they  were  entered,  so  that  extra 
attention  was  given  to  such  results.  This  may  have 
resulted  in  fewer  result  report  errors. 

Supervisors  at  all  three  sites  felt  that  duplication 
of  tests  was  reduced,  but  that  the  effect  was  likely 
small.  Supervisors  at  Oakland  estimated  that  there  might 
be  a  reduction  of  approximately  one  percent  in  total 
tests  because  of  the  improvement.  This  would  approximate 
a  reduction  of  about  450  tests  per  week  based  on  the 
current  workload.  Nursing  staff  estimated  that  40  or  50 
duplicated  tests  per  week  might  have  been  avoided.  To  be 
conservative,  it  was  estimated  that  1 00  duplicate  tests 
per  week  may  be  avoided,  representing  about  0.2  percent 
reduction  in  total  tests. 

In  addition  to  the  above  (quantifiable)  estimates  of  benefits, 
laboratory  staff  cited  the  following  benefits: 

•  Reduction  in  Transcription  Errors.  Because  abnormal 
results  were  highlighted  on  the  CRT  screens,  and  received 
extra  scrutiny  by  technicians  and  reviewers,  there  was 
potential  for  reduction  of  transcription  errors.  It  was 
felt,  however,  that  such  reduction  in  errors  was  likely 
small. 

•  Normal  Ranges  Data.  The  system  provided  the  capability 
to  provide  normal  ranges  data  with  each  test  (which  is  a 
CAP  accreditation  requires  -) .  This  may  not  have  been 
universally  provided  previously,  at  least  fof  the 
majority  of  tests  for  which  the  users  were  expected  to 
know  what  the  normal  ranges  were. 
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•  Search  Capability.  The  availability  of  the  computerized 
data  base  of  test  results  and  associated  demographic  data 
provides  the  potential  capability  of  performing  a  variety 
of  analyses  with  regard  to  utilization,  epidemiologic 
analysis,  etc.  This  capability  had  not  been  utilized  at 
the  time  of  the  post-implementation  study. 

•  Management  Reporting.  At  the  time  of  the  post¬ 
implementation  study  at  Oakland,  the  workload  reporting 
system  was  being  enhanced  to  provide  a  more  detailed 
analysis  of  workload  by  section,  shift  and  day  of  week, 
and  analysis  of  workload  per  assigned  (FTE)  staffing. 

This  would  enable  laboratory  management  to  improve  the 
allocation  of  staffing  resources  in  response  to  workload, 
and  thereby  improve  the  overall  efficiency  and  effective¬ 
ness  of  laboratory  services. 

b.  Problems 

Laboratorians  articulated  a  few  problems  that  they  have  had  with 
TRILAB,  one  of  them  being  the  response  time  of  the  system.  Some 
personnel  felt  that  terminal  response  time,  for  example  when  inputting 
corrections,  was  long,  and  others  felt  that  response  time  was  long  at 
peak  periods  of  the  day. 

Another  set  of  problems  mentioned  was  the  software  associated 
with  the  Microbiology  ("BACTI")  module,  which  did  not  have 
verification  or  "batch  entry"  features. 

2.  Benefits  to  Providers 

The  following  is  based  on  interviews  with  providers  (nursing 
staff  and  physicians)  at  the  three  sites. 

9  Reduced  Staff  Time.  Nursing  staff  at  all  three  sites 
estimated  that  there  was  considerable  reduction  in  staff 
time  associated  with  telephoning  the  laboratory  to 
receive  test  results  and  to  inquire  about  late  or  missing 
results,  in  filing  time  due  to  having  cumulative  reports 
available,  and  in  chart  review. 
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Staff  at  NRMC  Oakland  estimated  that  savings  amounted 
to  4.2  hours  per  inpatient  unit  or  clinic  (which  had 
terminals),  for  a  total  savings  in  the  hospital  of  75.6 
staff  hours  per  day.  Thus  time  savings  was  thereby  made 
available  for  other  activities.  At  Wright  Patterson, 
nursing  staff  estimated,  on  average,  savings  of  3.5  hours 
per  unit,  equivalent  to  49  hours  savings  in  staff  time  in 
the  hospital.  At  Eisenhower,  nursing  staff  estimated  an 
average  savings  of  3.7  hours  per  unit,  or  26  hours  per 
day  in  the  hospital. 

•  Duplicate  Tests.  Nursing  staff  at  the  three  ’  sites 
estimated  that  there  was  less  duplication  of  tests, 
perhaps  resulting  from  the  fact  that  results  appeared  on 
the  terminal  as  they  became  available,  and  there  was  less 
chance  they  would  be  lost.  It  was  felt,  however,  that 
such  reduction  was  small. 

•  Decreased  Turnaround  Time.  Turnaround  time  for  test 
results,  especially  for  routine  tests,  had  been  reduced, 
contributing  to  the  reduction  in  telephone  calls. 
Providers  indicated  that  this  may  have  resulted  in 
improved  patient  care. 

•  Improved  Morale.  As  a  result  of  being  able  to  look  up 
test  status  on  the  terminal,  and  the  reduction  in 
telephone  calls  to  the  laboratory,  nurses  felt  that 
relationships  between  nursing  and  laboratory  staff  had 
improved  considerably. 

•  Retrieval  of  Information.  Users  interviewed  relied 
heavily  on  TRILAB’s  information  storage  and  retrieval 
capabilities.  All  comments  in  this  regard  were  highly 
positive.  This  capability  was  reported  to  provide  a 
great  deal  of  information  to  users,  possibly  improving 
patient  care. 

•  Identification  of  Abnormals.  Because  abnormal  results 
were  identified  (by  an  asterisk),  leading  to  faster  and 
easier  identification  of  patient  problems,  providers  felt 
that  patient  care  has  been  improved. 
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To  summarize,  both  the  questionnaire  survey  (Section  C)  and  the 
interviews  indicated  that  health  care  providers  were  generally  pleased 
with  the  TRILAB  system,  citing  as  advantages  reduced  telephone  calls, 
decreased  test  turnaround  times,  improvements  in  relationships  with 
laboratory  personnel,  an  improvement  in  quality  of  care  due  to  easier 
and  faster  access  to  test  results,  identification  of  abnormal  values, 
and  cumulative  report  formats. 

A  further  indirect  measure  of  approval  of  the  system  was  the 
expressed  desire  of  staff  in  those  inpatient  units  and  outpatient 
clinics  that  did  not  have  terminals  (and  had  to  share  a  terminal  in 
another  location)  for  a  terminal  in  their  own  location. 

One  problem  that  was  expressed  fairly  uniformly  was  the  need  for 
additional  training  for  users,  particularly  administrative  staff 
(corpsmen).  It  was  felt  that  additional  hands-on  training  would  be 
particularly  useful,  in  addition  to  the  formal  (lecture)  type  of 
training. 
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IV.  COMPARISON  OF  RESULTS  WITH  PROJECT  OBJECTIVES 

On  July  15,  1977,  the  TRIMIS  Medical  Review  Group  (MRG)  developed 

(3) 

seven  project  objectives  for  the  Tri-Service  Laboratory  System  .  In 
this  section,  the  results  obtained  in  the  evaluation  are  related  to 
these  original  objectives. 

1.  To  Make  Information  Available  to  Physicians  with  Increased 
Efficiency  and  Accuracy 

Providers  reported  that  turnaround  time  for  test  results  had 
decreased  with  the  TRILAB  system;  this  was  confirmed  in  the  analysis 
of  turnaround  times.  The  turnaround  study  showed  that  for  routine 
tests,  results  were  available  to  provider  locations  in  less  time  for 
Chemistry  and  Hematology  tests,  and  in  about  the  same  time  period  or 
sooner  for  Microbiology .  For  STAT/urgent  tests,  process  times  were 
either  unchanged  (Chemistry)  or  reduced. 

Telephone  calls  to  the  laboratory  were  also  reported  to  occur 
with  less  frequency  under  the  operation  of  the  TRILAB  system  than  with 
the  previous  system,  as  well  as  tests  repeated  due  to  delays  or  to 
lost  results.  These  indicators  suggest  that  information  was  being 
made  available  to  physicians  with  increased  efficiency.  Data  were  not 
available  to  make  a  comparison  of  accuracy  of  results;  laboratory 
personnel  believed,  however,  that  accuracy  of  results  may  have  been 
improved  because  of  highlighting  of  unusual  values  by  the  system, 
facilitating  review  of  such  results.  Most  providers  felt,  however, 
that  repeating  of  tests  due  to  inaccurate  results  occurred  with 
similar  frequency  with  the  TRILAB  system  as  before. 

Providers,  especially  those  on  inpatient  services,  were  very- 
satisfied  with  the  ability  of  retrieving  patient  laboratory  results 
via  terminal  inquiry,  and  with  the  cumulative  results  reports.  In 
some  cases  the  cumulative  reports  were  used  to  supplement  the  manual 
"flow  sheets,"  and  in  others  had  replaced  them.  Nursing  personnel  at 
all  three  sites  estimated  that,  on  average,  approximately  four  staff 
hours  per  day  had  been  saved  at  each  inpatient  unit  or  outpatient 
clinic  which  had  a  terminal,  through  reductions  in  time  on  the 
telephone,  filing  time,  and  chart  review. 
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2.  To  Present  the  Data  in  a  Convenient  and  Meaningful  Manner 


with  Sufficient  Variety  in  Report  Formats  to  Meet  the  Needs 
of  All  Users 

Providers  reported  that  they  were  very  satisfied  with  the  report 
formats  provided  by  the  TRILAB  system.  Both  physicians  and  nursing 
staff  indicated  significant  increase  in  their  satisfaction  with 
legibility  of  laboratory  reports  between  the  two  study  periods. 

3.  To  Be  Able  to  Handle  Increased  Demands  for  Laboratory  Testing 
Without  Significant  Increases  in  Staff 

The  evaluation  results  indicate  that  less  time  of  laboratory 
staff  was  being  devoted  to  clerical  activities,  such  as’  workload 
reporting,  quality  control  reporting,  and  transcription  and  recording 
of  test  results,  on  telephone  calls,  and  in  number  of  staff  required 
at  the  reception  area.  At  Oakland,  the  percent  of  time  devoted  to 
processing  of  test  results  increased  by  a  (modest)  1.2  percent.  These 
results  suggest  that  the  TRILAB  system  will  improve  the  ability  of 
laboratory  personnel  to  handle  increased  demands  without  significant 
increases  in  staff. 

4.  To  Provide  Accountability  of  Laboratory  Requests  and  To  Monitor 
Generation  of  Test  Results  to  Include  Providing  Notices  of 
Abnormal  Values  or  Improper  Quality  Control  Results  as  Soon  as 
They  are  Available 

The  system  provides  immediate  highlighting  of  abnormal  values  and 
unusual  changes  from  previous  results,  facilitating  review  by 
laboratory  personnel  and  pathologists.  This  objective  is  therefore 
considered  to  have  been  met. 

5.  To  Gather  as  a  Result  of  Normal  Procedures,  Workload  and 
Managerial  Data,  and  to  Present  This  as  Required  in  Order 
to  Assist  in  Decision-Making  in  the  Laboratory 

The  system  gathered  and  presented  workload  data  (thereby  reducing 
the  time  devoted  to  this  activity) .  The  workload  reports  were  being 
modified  to  enhance  their  capability  to  provide  workload  and 
managerial  data,  which  should  lead  to  improved  capability  of 
allocating  laboratory  resources  in  response  to  required  workload. 
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6.  To  Reduce  the  Clerical  Work  Required  of  Qualified  Technicians 
in  the  Laboratory 

The  work  sampling  results  at  Oakland  indicate  that,  overall,  time 
devoted  to  information  handling  activities  was  about  6  percent  less  in 
the  post-implementation  period  compared  with  the  baseline  period;  this 
difference  was  about  2.6  percent  of  total  laboratory  technician  staff 
time.  Laboratory  personnel  at  all  three  sites  indicated  that 
identification  of  incomplete  and  pending  tests  and  results  was 
improved  by  the  TRILAB  system  and  that  time  spent  on  manual 
record-keeping  occurred  less  frequently  with  the  TRILAB  system.  They 
reported  that  "efficiency  of  laboratory  operations"  and  "ease  of 
information  storage  and  retrieval"  were  either  "very  important"  or 
"somewhat  important"  improvements  due  to  TRILAB. 

The  major  change  in  non-personnel  operating  costs  in  the 
laboratory  was  likely  due  to  a  reduction  in  duplicated  tests,  and  the 
associated  reagent  costs.  Providers  and  laboratory  staff  at  Oakland 
estimated  that  the  reduction  might  have  been  approximately  100  tests 
per  week. 

7,  To  Improve  Result  Accuracy  by  Eliminating  Transcription, 
Calculation,  and  Specimen  Identification  Error 

No  data  were  available  to  measure  this  effect.  Providers  and 
laboratory  staff  at  the  three  sites,  however,  believed  that  such 
errors  have  been  somewhat  reduced,  resulting  in  fewer  duplicate  tests 
required. 

It  is  concluded  that  the  original  project  goals  have  by  and  large 
been  met,  albeit  modestly  for  the  objectives  of  enabling  the 
laboratory  to  handle  increased  demands  for  testing  and  for  reducing 
the  clerical  work  required  by  laboratory  technicians. 
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